Multisensory Design

Multisensory Design Research at The National College of Art & Design ©

May 1, 2011

Key words: sensory modal switching, eight senses, product experience, Multisensory Response, sensory anthropology, memory layering.

Abstract

“Sensory Modal Switching” (SMS) is a new and exciting paradigm derived by the MSD Group at NCAD. SMS occurs spontaneously and maintains a continuous flow of modal activity. In the course of forming a perception, we interpret external objects through all of our senses. The brain executes a straightforward process of sensing an external stimulus, transmitting the sensations through the nervous system to the brain for identification and evaluation in relation to context and stored memory. When a person chooses to examine an object through a particular sensory modality, this simple process is replicated many times over as the brain responds by modulating the conscious impression with cross-modal integration of the remaining senses. Furthermore, the brain immediately follows up by involuntarily “switching” between all the remaining senses in turn with related cross-modal activity supporting each succeeding sensory modality until the perception is completed. The result is an ever expanding layering effect within our memory. SMS is the most important element in a new concept within multi-sensory processing that has emerged from our research which enables us to focus on the fact that a person perceives an external object by investigating all sensory aspects of its composition and not merely by means of only one or two sensory modalities. This novel concept opens up new opportunities for designers that will be explored in follow-up papers in this series.

Introduction

We at the MSD Group are seeking a means of encouraging product designers to create designs that stimulate the complete range of sensory modalities in the user. We hope to draw the attention of designers and users alike to the fact that our newly discovered paradigm of “Sensory Modal Switching” is the mechanism that product users avail of to explore the sensory aspects of a design and create a Multisensory Response Pattern.

We seek to answer the following questions: what is the extent of sensory information we store in memory? And how is the sensory information received as a message?

In search of an answer, our studies of secondary sources have taken us through Sensory Modality, Cross Modal Perception, Sensory Anthropology, Psychology, Multisensory Processing, Human Cognition & Affordances and Neuroscience. These disciplines have greatly increased our understanding of the sensory functions related to perception. We look forward to joining in with the many eminent scholars who are active in the field and to making our own contribution to this research.

This is the first in a series of discussion papers that are intended to generate debate amongst scholars, designers and all interested persons. All contributions are welcome! Future papers in this series will explore the theory that designers who deliver enough sensory information through all the modalities will produce products that enable Sensory Modal Switching.

Multisensory Processing

Scientists have identified a staggering total of 28 separate senses. These are grouped into external and internal senses. The external (exteroceptive) senses cover eight senses including vision, touch, hearing, smell, taste, balance, body awareness and temperature. The internal (interoceptive) senses come from the action of internal organs and nerves and give important information to the brain about the body’s reaction to the object under scrutiny. (Sense, 2011)

Our study deals with the external senses which people use to examine a product. There is evidence that multisensory integration effects play a key role in drawing a person’s attention to different aspects of an external object (Calvert, Spence, & Stein, 2004, p. 425) (Schifferstein & Hekkert, 2008, p. 1). All the senses are permanently active as the user approaches an object in search of sensory information, checking and re-checking to confirm the external stimulus as a real object (Noë, 2004, p. 6). As the search progresses, the focus of attention shifts from one sense to the other, seeking clear, recognizable information about the perceived object (Shimojo & Shams, 2001, pp. 505–509). This is done simply to identify the object and to associate it with a known experience in memory. Experiences that may at first appear to be modality-specific are most likely to have been influenced by activity in other sensory modalities despite our lack of awareness of such interactions (Calvert, Spence, & Stein, 2004, p. XI).

The sensory modality that is temporarily in focus is in a dominant relationship to all of the remaining senses for that moment in time. When one sense is chosen as the dominant means of perception, the other senses initiate a new sequence to supply additional supporting information. The proportion of attention given to each of the remaining senses is continuously raised or lowered and re-checked as the brain detects the presence of new information coming in through the entire nervous system. The brain filters out information from the supporting senses that does not relate to the information coming from the dominant modality (Ernst & Bulthoff, 2004, p. 1). A rich, fluid tapestry of sensory information comes from a combination of external and internal sensory processes that interact with one another in continuous, energetic ways that are unique to the individual. The central force of this energy is the unyielding search for a sensory stimulus. This basic energy flow is replicated for each of the multiple sensory events that occur during a product experience so that each event is “detected rapidly, identified correctly and responded to appropriately”. As James J. Gibson has expressed it, “there is a loop from response to stimulus to response again and the result may be a continuous flow of activity rather than a chain of distinct reflexes” (Gibson, 1966, p. 31).

Layering within Memory

We consciously switch our focus of attention between dominant modalities for a variety of reasons including being creative, a desire for alternative experiences or to replicate an old one, to engage with a sensory opportunity, to focus attention, etc. Thus, we seize the opportunity to play with our world and explore it in different ways. The brain has a voracious appetite for experiences and relies upon the sensory modalities to satisfy its hunger (Schifferstein & Spence, 2008, p. 137). This hunger keeps the senses constantly active, looping in search of external stimuli. Until recently, conventional wisdom held that the brain entered a state of hibernation while sleeping and that the senses could be activated only by outside stimuli. This has been superseded by a new paradigm “in which the brain is constantly active and stimuli change and shape that activity” (Vieru, 2009). Therefore, the sensory impression associated with any given object is a work in progress and never fully complete. Each experience results in the creation of a bespoke and event-based perception that is “modulated by many contextual factors such as multisensory information, past experiences, internal predictions, associations, on-going motor behaviour, spatial relations and the nature of the task itself” (Calvert, Spence, & Stein, 2004, p. 135).

Bergsen has expressed it as follows “there is no perception which is not full of memories. With the immediate and present data of our senses, we mingle a thousand details out of our past experience” (Bergsen, 1988, p. 24).

We suggest that when there is no discernable stimulation the brain supplies the results of previous sensory experiences from memory associated with observation; described by Newell as “ the object, material, shape, colour or image” (Newell, 2004, p. 123). We call this “ghosting” and the brain does this in order to complete the “sensory message”. A true object identity or association can only be established with memory when the sensory message is “completed” by providing eight separate sensory values to the object. Additional associated reference signatures such as time lapse, context, cause and effect, a whole body experience, good or bad ruling, avoidance or seeking rules and numerous others including emotional feelings, enrich the impression and create multiple points of reference for future matching. Furthermore, we must not underestimate Howes’ observation concerning the ability of culture, society and history to influence our sensory associations (Howes, 2005, p. 4).

This begs the question: Does our memory influence our future sensory experiences?

Perhaps “Adaptive Resonance Theory” (ART), developed by Stephen Grossberg and Gail Carpenter to explain how the brain processes information, can provide an answer. The primary intuition behind the ART model is that object identification and recognition generally occur as a result of the interaction of observer expectations with sensory information. The model postulates that expectations take the form of a memory template or prototype that is then compared with the actual features of an object as detected by the senses. This comparison gives rise to a measure of category belongingness. As long as this difference between sensation and expectation does not exceed a set threshold called the 'vigilance parameter', the sensed object will be considered a member of the expected class within memory” (Arbib, 2003, pp. 87-90).

A similar theory: “the Maximum-Likelihood Integration Model” hypothesized by Ernst and Banks states that the observer determines the modality weights by operating as a maximum-likelihood integrator to minimize uncertainty (cf. full presentation in (Calvert, Spence, & Stein, 2004, p. 119). Bayes’ Probability Theory also elaborates this connection with memory as a tool to provide us with certainty as to what we are sensing (Ernst & Bulthoff, 2004, p. 164).

Both of these approaches acknowledge that the brain needs to create matching points of reference and to provide pre-defined parameters for this function. We find these theories fascinating, but would feel they need to be expanded upon to include a sensory modal switching process which enables the cross checking to happen.

Sensory Modal Switching

Sensory Modal Switching is a new paradigm that has been devised by the MSD Group to explain how the brain employs all of our sensory modalities in the course of completing a perception. It is an intuitive sensory process practiced unconsciously by us all.

The principal triggering mechanism for this process is our brains’ voracious appetite for experiences that can be cross checked within our memories vigilance parameters. This need must be satisfied in order to create a state of “knowing”. Not knowing would plunge us into a state of fear and anxiety and thus its core function is to provide comfort.

When faced with an object of interest, the user intuitively switches through every sensory modality in turn, testing all known cross modal combinations, until satisfied that all aspects of the designs sensory message are understood and associated with memory. This cyclical checking of combinations is what we are calling “Sensory Modal Switching”. Numerous sensory combinations are checked in rapid sequence to generate an adequate flow of information. The process is repeated continuously at the same rapid pace until the user is satisfied that all available sensory information has been found and compared within a vigilance parameter. We propose that the brain may provide each sequence and combination with a signature which is remembered independently and stored in our unconscious mind. We feel justified in presuming that there are, at least twenty-eight sensory modalities and their associated combinations of same which make up the completed “switching” menu. This switching process plays a crucial role in enabling a whole body experience to be created in memory and ensures future cross modal sensory connections can be replicated and improved upon. We pose that momentary events such as “context change” are also recorded in memory and associated with sensory input within that event. Perceptions are updated, built upon and duplicated within memory groupings to ensure correct future associations. All of the contributing factors involved in an experience affect the switching mechanism.

Thus each memory created describes a component or “momentary occurrence” within the continuous process of Sensory Modal Switching. The benefit of Sensory Modal Switching is that it increases the accuracy of the senses to interpret the external object. The understanding of this control mechanism within multisensory processing is the key to creating a richer product experience.

Conclusions, and direction for continuing research

In this paper we have discussed multisensory processing, layering in memory and sensory modal switching as the essential processes involved in multisensory perception. Sensory Modal Switching is a new paradigm derived by the MSD Group to explain how the sensory system controls the continuous flow of modal activity. We have shown that each person arrives at a perception in accordance with their own unique, highly subjective, dynamic, flexible and adaptive approach. From time to time, a person may choose to focus their conscious attention on the information coming to them through a particular sensory modality. When they do so, Sensory Modal Switching begins searching for supportive information from the remaining senses to help clarify the information coming from the dominant sense without the individual necessarily being aware of it.

The nature of the activity is unique for that point in time and in relation to that particular observation and context. As soon as a new object comes to the attention of the individual or if the context changes, the whole process starts up again and may well come to rest on a different sensory modality at the end of its search. These choices are personal and intuitive. Therefore it is not possible to predict which sensory modality will be favoured by an individual in the course of building a perception. It is true that research instruments can measure each sensory modality but the results can only describe the proportion of user attention given to each of the sensory modalities during one experience and at one moment in time. An understanding of the nature of these processes is essential for profiling an individual’s perception.

This view is reinforced by the observations concerningcross modal interactions (Shimojo & Shams, 2001), Adaptive Resonance Theory (Arbib, 2003) and cognitive style (Kozhevnikov, 2007, pp. 464-481). The development of a person’s multisensory perception is an on-going and never-ending process which continually references and confirms memories formed through life, enhanced and moulded by their social and cultural history (Howes, 2005).

Likewise, emotional factors which influence the way in which we engage with our world, such as alertness, stress, tiredness, fatigue, concentration, happiness, excitement, etc., all trigger us to change priorities within the Sensory Modal Switching process and thus affect our multisensory perception. The ability to switch sensory modalities and adapt to our environmental context within an instant is a feature of the flexibility of our multisensory perception and the reason for the success of the human species.

Our research is fuelled by a firm belief that product perception comprises information received from all the senses. We also believe that the user completes the design message unconsciously and intuitively, ensuring that it comprises information for all eight sensory modalities. Sensory Modal Switching as a paradigm opens up exciting prospects for greater understanding of the multisensory relationship between the end user and the output of product design.

New research is needed to explore sensory modal switching, its impact on multisensory processing and its consequent effect on memory.

Forthcoming papers will discuss the ramifications of this new paradigm and the interesting prospect of its relationship to the layering of memory over time.

We look forward with interest to a healthy debate on these questions and invite all our readers to join in to ensure that the widest possible range of views are brought to bear on these new ideas.

Acknowledgements

The MSD group has been conducting research into multisensory perception since May ‘08 at the National College of Art and Design (NCAD), Dublin, Ireland. Our primary concern has been to enhance design processes so that users derive the greatest experience from products. Accordingly, the MSD research team has focussed on the senses together with the physiological and neurological processes involved in product interaction.

Many have entered this field of research before us and their findings have provided a solid base on which we have been able to build our own research. In particular, we found encouragement in the published work on multisensory design and product experience led by Dr. Henrik J. Schifferstein of TU Delft University, The Netherlands. We found much inspiration in the work of Professor David Howes of Concordia University, Canada, who has conducted important research into the cultural aspects of sensory perception, and are grateful for the interest he has shown in our efforts. The Handbook of Multisensory Processes has of course proven to be a goldmine of useful information. The published work of Professor Alva Noé at the University of California USA proved an invaluable source of information on the significance of bodily action on perception. Many other scholars have in their published work raised issues that seem to confirm our standpoint and compel us to dig deeper and develop our argument.

This paper, based on secondary sources, concerns one aspect of a larger ongoing research project. We are particularly grateful to the NCAD for providing the facilities to make this research possible. We are indebted to the Bank of Ireland for their financial support during the first two years of the project. We wish to thank undergraduate assistant Steven O’Toole of NUI Maynooth who executed such thorough research that the project moved from a possible idea to reality, of which the present offering forms the first publication, and also to Rebecca Matthews, research assistant from NCAD, for her assistance with the initial four months of the project.

References

Cutaneous detection of movement. (2009). Retrieved 06 30, 2009, from Informa Health care: http://informahealthcare.com/doi/abs/10.3109/14992029709048007

feeling of movement . (2009). Retrieved 06 30, 2009, from TheFreeDictionary : http://www.thefreedictionary.com/feeling+of+movement

Balance (ability). (2010). Retrieved 06 30, 2010, from Wikipedia, the free encyclopedia : http://en.wikipedia.org/wiki/Balance_(ability)

Equilibrioception. (2010). Retrieved 06 30, 2010, from Wikipedia, the free encyclopedia: http://en.wikipedia.org/wiki/Equilibrioception

Exteroceptive. (2010). Retrieved 06 30, 2010, from The Medical Dictionary,: http://medical-dictionary.thefreedictionary.com/exteroceptive

Hearing (sense). (2010). Retrieved 06 30, 2010, from Wikipedia, the free encyclopedia : http://en.wikipedia.org/wiki/Hearing_(sense)

Olfaction. (2010). Retrieved 06 30, 2010, from Wikipedia, the free encyclopedia : http://en.wikipedia.org/wiki/Olfaction

Proprioception. (2010). Retrieved 06 30, 2010, from Wikipedia, the free encyclopedia : http://en.wikipedia.org/wiki/Proprioception#Proprioception_vs._kinesthesia

Sense. (2011). Retrieved 05 10, 2011, from wikipidea,the free encyclopedia: http://en.wikipedia.org/wiki/Sense

Sensory System,(2010). Retrieved 04 10, 2010, from Wikipedia, the free encyclopedia: http://en.wikipedia.org/wiki/Sensory_system

Somatosensory system. (2010). Retrieved 06 30, 2010, from Wikipedia, the free encyclopedia : http://en.wikipedia.org/wiki/Touch

Taste. (2010). Retrieved 06 30, 2010, from Wikipedia, the free encyclopedia: http://en.wikipedia.org/wiki/Gustation

Vestibular system. (2010). Retrieved 01 04, 2011, from Wikipedia, the free encyclopedia : http://en.wikipedia.org/wiki/Vestibular_system

Visual perception. (2010). Retrieved 06 30, 2010, from Wikipedia, the free encyclopedia : http://en.wikipedia.org/wiki/Visual_perception

Arbib, M. A. (Ed.) ( 2003). The Handbook of Brain Theory and Neural Networks (2 nd ed.). Cambridge, Mass.: MIT Press.

Bergson, H. (1988). Matter and Memory (trans. N.M. Paul and W.S. Palmer; 2nd ed.). New York: Zone Books.

Ernst, M.O. & Bulthoff, H.H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences 8(4), 162-169.

Calvert, G. A., Spence, C., & Stein, B. E. (Eds., 2004). The Handbook of Multisensory Processes.). Cambridge Mass:MIT Press.

Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal way. Nature, 415, 429–433.

Gibson, J. J. (1966). The Senses Considered as Perceptual Systems. Boston: Houghton-Mifflin.

Howes, D. (Ed.) (2005). Empire of the Senses : The Sensual Culture Reader. Oxford and New York: Berg Publshers

Kozhevnikov, M. (2007). Cognitive Styles in the Context of Modern Psychology. Psychological Bulletin of The American Psychological Association, Vol. 133(No. 3), 464–481.

Newell, F. N. (2004). Cross-Modal Object Recognition. In Calvert, G. A., Spence, C., & Stein, B. E. (Eds.) (2004), The Handbook of Multisensory Processes. Cambridge Mass: MIT Press.

Schifferstein, H.N.J. & Hekkert, PPM. (Eds.) (2008). Product Experience. Elsevier Science

Schifferstein, H.N. J., & Smeets, M. A. (2006). Towards the assessment of perceptual style, In M. Karlsson (Ed.) Proceedings of the 5th International Conference on Design and Emotion [CD ROM]. Gothenburg: Chalmers University Press.

Schifferstein, H.N.J-. & Spence, C. (2008). Multisensory product experience. In Schifferstein H.N.J.& Hekkert P.P.M. (Eds.), Product Experience (pp. 133-161). Amsterdam: Elsevier.

Scott 1, L. (2011). Why do birds sing in the morning? Retrieved 01 06, 2011, from The RSPB Advice: http://www.rspb.org.uk/advice/expert/previous/morningsong.aspx

Scott 2, L. (2011). We can hear birds singing at night . Retrieved 01 06, 2011, from The RSPB Advice : http://www.rspb.org.uk/advice/expert/previous/singingatnight.aspx

Shimojo, S., & Shams, L. (2001). Sensory modalities are not separate modalities: plasticity and interactions. Current Opinion in Neurobiology, 11 (114):505–509.

Spence, C., & Driver, J. (1997). Audiovisual links in exogenous covert spatial orienting. Perception & Psychophysics, 59, 1-22.

Sternberg, R. J. (1985). Beyond IQ: A Triarchic Theory of Intelligence. Cambridge: CUP.

Vieru, T. (2009). Your-Brain-Never-Sleeps. Retrieved April 05/04/11, 2011, from Softpedia: http://news.softpedia.com/news/Your-Brain-Never-Sleeps-103789.shtml